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The  most  operationally  realistic  testing  of  chemical  and  biological  defense  systems  uses  actual 
biological  and  chemical  warfare  agent;  however,  testing  live  or  actual  agents  is  restricted  to 
laboratory  containment  chambers,  which  are  operationally  unrealistic  environments.  This  state 
of  affairs  has  driven  the  chemical  and  biological  defense  community  to  integrate  developmental 
testing  (DT)  and  operational  testing  ( OT)  to  support  the  evaluation  process.  Three  paradigms 
are  commonly  used  to  integrate  chemical  and  biological  defense  DT  and  OT.  They  are  (a) 
conducting  DT  with  systems  before  and  after  OT,  (b)  developing  agent  simulant  relationships 
in  DT,  which  are  then  applied  to  OT  data,  and  (c)  modeling  and  simulation.  This  article 
supplies  chemical  and  biological  defense  system  examples  for  each  paradigm. 
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integrated  testing;  modeling  and  simulation;  operational  testing;  public  safety;  realistic 
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The  most  operationally  realistic  testing 
of  chemical  and  biological  defense 
systems  will  use  actual  biological  and 
chemical  warfare  agent.  As  a  result  of 
public  law,  treaty,  and  concern  for 
public  safety,  testing  with  actual  agent  is  restricted  to 
laboratory  containment  chambers,  which  is  the  least 
operationally  realistic  environment.  As  a  result,  the 
chemical  and  biological  defense  test  and  evaluation 
(T&E)  community  has  been  forced  to  develop  and  use 
paradigms  that  combine  operational  testing  and 
developmental  testing.  The  operational  test  (OT)  part 
brings  the  realism  of  actual  warfighters  executing 
missions  in  combat-like  environments.  The  develop¬ 
mental  test  (DT)  part  brings  the  realism  of  actual 
biological  or  chemical  agent.  These  paradigms  are  the 
following: 

•  conducting  DT  with  systems  before  and  after 
OT, 

•  developing  agent  simulant  relationships,  and 
•  modeling  and  simulation. 

A  simulant  is  a  relatively  harmless  substance  that  has 
some  of  the  properties  of  agents  and  can  be  released 
into  the  environment.  These  three  basic  paradigms  are 


not  mutually  exclusive  and  may  be  used  in  combination 
on  the  same  system  under  test. 

This  article  describes  how  these  three  basic  para¬ 
digms  to  integrate  DT  and  OT  have  been  used  as  the 
lynch  pins  for  evaluation  of  chemical  and  biological 
defense  systems. 

Paradigm  1 :  DT  with  systems  before  and 
after  OT 

During  OT,  systems  are  used  by  warfighters  while 
executing  wartime  missions.  Use,  care,  and  mainte¬ 
nance  of  the  system  under  test  are  typical  of  what 
could  be  expected  during  actual  operations.  OT,  like 
wartime,  tends  to  be  a  strenuous  environment. 
During  OT,  system  performance  is  often  degraded. 
The  total  effect  of  an  OT  on  the  performance  of 
system  under  test  can  be  determined  by  conducting 
DT  performance  testing  on  those  systems  both 
before  and  after  the  OT.  The  effect  of  less  use  than 
is  measured  by  a  whole  OT  on  the  performance  of  a 
system  under  test  can  be  measured  by  periodically 
removing  systems  from  OT  for  DT  performance 
testing.  This  paradigm  has  been  routinely  used  on 
protective  garments  since  the  Joint  Service  Light¬ 
weight  Integrated  Suit  Technology  (JSLIST)  testing 
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Figure  1.  Beach  assault  from  Joint  Service  Lightweight  Integrated  Suit  Technology  operational  testing. 


in  the  early  1990s.  It  was  also  applied  to  the  Joint 
Biological  Point  Detection  System  (JBPDS)  as  well 
as  other  detectors. 

During  OT,  the  JSLIST  candidate  protective 
garments  were  worn  by  U.S.  Marines,  soldiers,  airmen, 
and  sailors  while  conducting  combat  missions.  These 
missions  included  offensive  and  defensive  actions  at  29 
Palms  Marine  Corps  Base,  a  beach  assault  at  Camp 
Pendleton  (Figure  1),  shipboard  operations,  and  flight 
line  operations.  The  missions  were  intended  to  put  the 
same  wear,  stresses,  and  strains  on  JSLIST  as  would 
occur  during  combat.  The  garments  were  worn  for  up 
to  60  days.  They  were  worn  23  hours  a  day. 
Warfighters  not  only  conducted  missions  in  JSLIST 
but  also  slept  and  ate  in  this  protective  garment.  The 
suit  was  laundered  every  other  week.  A  random  sample 
of  JSLIST  protective  garments  and  the  base  line 
protective  garment  were  removed  from  the  OT  after 
15,  30,  45,  and  60  days  of  wear  and  sent  to  DT 
performance  testing.  It  was  known  that  mission  types 
were  not  equivalent.  Offensive  actions  result  in  more 
wear  on  garments  than  defensive  operations.  Hence, 
care  was  taken  to  randomly  choose  garments  and  to 
spread  mission  types  somewhat  uniformly  over  the  60 
days  of  OT. 

The  DT  performance  testing  include  various  types 
of  swatch  testing  with  various  chemical  warfare  agents, 
and  whole  system  testing  in  which  warfighters  wore 
JSLIST  in  a  simulant  chamber  performing  a  prescribed 
task  list.  In  addition  to  the  garments  with  15,  30,  45, 


and  60  days  of  wear,  new  garments  were  also  tested  in 
DT  performance  testing. 

The  JSLIST  evaluation  compared  the  effect  of 
mission  wear  time  on  the  performance  degradation  of 
JSLIST  candidates  to  the  baseline.  The  fielded 
protective  garment  served  as  the  baseline.  This  strategy 
of  performing  DT  both  before  and  after  OT  resulted  in 
identifying  the  JSLIST  candidate  garment  that  best 
increased  the  amount  of  protection  (Musgrave  et  al. 
1997). 

The  JBPDS  was  tested  in  accordance  with  this 
paradigm  in  the  Ambient  Breeze  Tunnel  (ABT) 
before  and  after  OT  1.  The  ABT  is  a  DT  wind  tunnel 
in  which  simulant  concentration  can  be  accurately 
controlled  and  measured.  This  testing  revealed  that 
after  OT  there  was  significant  and  substantial 
degradation  in  detection  performance.  The  degrada¬ 
tion  in  performance  was  traced  to  a  LASER  in  the 
detection  system.  The  LASER  design  was  upgraded. 
Subsequent  testing  demonstrated  that  the  new 
LASER  did  not  degrade  during  OT  (Chipman  et 
al.  2001). 

This  paradigm  combines  or  integrates  DT  and  OT 
results  to  produce  a  powerful  evaluation  tool  to  ascertain 
if  system  performance  is  degraded  during  OT. 

Paradigm  2:  Agent-simulant  relationships 

A  simulant  is  a  relatively  harmless  substance  that 
possesses,  for  the  attributes  being  measured  or  tested, 
properties  similar  to  agents  of  interest  and  that  may  be 
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Figure  2.  S-shaped  or  sigmoid  curve  depicting  the  reiationship  between  agent  detection  and  agent  concentration. 


released  into  the  environment  since  the  simulant  is  not 
dangerous.  Much  effort  is  invested  in  simulant  selection 
and  development.  For  testing  detectors,  it  is  desirable 
that  the  system  under  test  “sees”  the  simulant  the  same 
way  as  it  “sees”  the  agent  that  the  simulant  is  simulating. 
Other  properties  of  interest  include  similarities  in 
dissemination  and  cloud  dynamics.  No  simulant  is  an 
exact  match  in  all  properties  of  interest  to  the  agent. 

Traditional  biological  detection  simulants  include 
ovalbumin  for  toxins,  the  bacteriophage  Male  Stereo¬ 
type  2  (MS2)  for  viral  biological  warfare  agents,  Erwinia 
herbicola  for  vegetative  biological  warfare  agents,  and 
Bacillus  subtilis  for  spore-forming  biological  warfare 
agents.  Chemical  simulants  are  much  more  numerous 
than  biological  simulants;  two  commonly  used  simulants 
for  chemical  detectors  are  triethyl  phosphate  for  nerve 
agent  and  acetic  acid  for  blister  agent. 

The  Biological  Integrated  Detection  System  (BIDS) 
and  BIDS  Pre-Planned  Product  Improvement  (P3I) 
were  tested  in  the  early  to  mid  1990s.  They  used  hoth 
biological  warfare  agent  and  the  traditional  biological 
detection  simulants  in  DT  laboratory  containment 
chamber  testing  and  traditional  biological  detection 
simulants  in  OT.  Inferences  focused  on  laboratory 
results  with  agent  and  OT  results  with  simulant.  It  was 
noted  in  DT  that  the  detection  performance  of  BIDS 
for  a  group  of  agents  lay  between  the  values  of  two  of 
the  stimulants.  This  finding  enabled  us  to  predict  that 
the  field  performance  of  BIDS  against  these  agents 
would  be  bounded  by  the  OT  performance  against 
these  two  simulants. 


For  both  biological  warfare  agent  detectors  and 
chemical  warfare  agent  detectors,  the  variable  that  has 
the  most  profound  effect  on  detection  performance  is 
agent  concentration.  As  concentration  increases, 
detection  performance  tends  to  increase.  At  some 
high  level  of  agent  concentration,  all  releases  are 
detected.  As  concentration  decreases,  detection  per¬ 
formance  tends  to  decrease.  At  some  low  level  of 
agent  concentration,  no  releases  are  detected.  The  plot 
formed  by  agent-concentration  and  detector-perfor¬ 
mance  is  s-shaped  or  sigmoid  as  is  depicted  in 
Figure  2.  Logistic  regression  is  a  statistical  framework 
based  on  a  sigmoid  relationship.  Logistic  regression 
has  been  useful  in  modeling  detection  performance 
and  in  developing  the  agent- simulant  relationships  in 
both  the  JBPDS  and  the  Joint  Chemical  Agent 
Detector. 

Three  of  the  traditional  biological  detection  simu¬ 
lants — ovalbumin,  MS2,  and  Erwinia  herbicola — have 
been  considered  inadequate  representations  of  actual 
biological  warfare  agent  (Fitch  et  al.  2004).  In  an  effort 
to  obtain  simulants  that  better  represent  biological 
warfare  agents  of  interest,  closely  related  organisms  or 
vaccine  strains  were  used  for  viruses  and  bacteria. 
Toxoids  were  used  for  toxin.  A  toxoid  is  a  toxin  that 
has  been  denatured  or  broken  into  nonhazardous 
components.  The  toxoids,  closely  related  organisms, 
and  vaccine  strains  are  referred  to  as  agent-like 
organisms  (ALO).  To  eliminate  any  chance  of 
infection  from  the  closely  related  organisms  or  vaccine 
strains,  these  were  killed  with  ionizing  radiation. 
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The  DT  portion  of  the  Whole  System  Live  Agent 
Test  of  JBPDS  was  conducted  to  test  JBPDS  detection 
performance  when  challenged  with  biological  warfare 
agent  and  ALOs.  Both  live  and  killed  agent  and  ALO 
were  used.  JBPDS  performance  testing  was  conducted 
with  both  living  and  killed  ALOs,  and  living  and  killed 
agents,  in  laboratory  containment  chambers.  JBPDS 
performance  testing  was  conducted  with  only  killed 
ALO  simulants  in  the  ABT  and  field.  The  end  results 
of  this  DT  testing  were  twofold: 

1.  JBPDS  detection  performance  when  challenged 
with  live  agent  in  a  laboratory  containment 
chamber  was  characterized. 

2.  Relationships  were  developed  to  relate  JBPDS 
detection  performance  when  challenged  with 
killed  ALO  simulants  to  JBPDS  detection 
performance  when  challenged  with  live  agent. 

The  JBPDS  OT  used  the  new  killed  ALO  simulants. 
The  relationships  developed  in  DT  between  JBPDS 
detection  performance  with  killed  ALO  simulants  and 
live  agent  were  used  to  predict  JBPDS  performance  as 
if  it  had  been  challenged  with  live  agent  (Holman  et  al. 
2008). 

The  evaluation  of  JBPDS  provided  predictions  of 
JBPDS  performance  against  live  agent  based  upon  OT 
challenges  with  killed  ALO  simulants  and  relation¬ 
ships  developed  in  DT. 

Paradigm  2,  the  development  agent-simulant  rela¬ 
tionships,  combines  or  integrates  relationships  devel¬ 
oped  in  DT  with  OT  results  to  support  an  integrated 
evaluation  of  how  the  system  performs  against  agent. 

Paradigm  3:  Modeling  and  simulation 

The  keystone  in  both  the  Joint  Service  Lightweight 
Standoff  Chemical  Agent  Detector  QSLSCAD)  and 
the  Joint  Biological  Standoff  Detection  System 
0BSDS)  evaluations  was  the  use  of  modeling  and 
simulation  to  integrate  DT  and  OT  results  to  predict 
system  detection  performance. 

JSLSCAD  modeling  and  simulation  was  conducted 
by  a  team  that  included  Johns  Hopkins  University 
Applied  Physics  Laboratory  (JHU  APL),  the  Joint 
Project  Manager  for  Contamination  Avoidance,  and 
both  Dugway  Proving  Ground  and  the  U.S.  Army 
Evaluation  Center  from  the  Army  Test  and  Evaluation 
Command.  The  backbone  of  the  JSLSCAD  model  was 
a  model  developed  by  Honeywell,  which  was  used  to 
support  JSLSCAD  development.  Spectral  backgrounds, 
meteorological  data,  and  other  data  were  collected  by 
JHU  APL  in  various  locations  of  tactical  interest  and 
used  in  the  JSLSCAD  model.  The  vapor,  liquid,  and 
solid  tracking  transport  and  dispersion  model,  with 
chemical  and  biological  analyzer,  was  used  to  provide 


cloud  size,  concentration,  and  cloud  propagation 
information  for  chemical  warfare  agents.  A  scanning 
model  was  used  to  allow  the  JSLSCAD  model  to  have 
multiple  chances  of  detecting  a  chemical  cloud  as  it 
moved  downwind.  OT  and  DT  field  tests  and 
laboratory  tests  provided  a  base  for  validation,  verifica¬ 
tion,  and  accreditation  of  the  modeling  and  simulation. 

The  modeling  and  simulation  results  were  the  best 
source  of  information  on  the  detection  performance  of 
JSLSCAD.  The  modehng  and  simulation  provided  a 
means  to  integrate  DT  and  OT  in  an  evaluation  that 
provided  information  on  the  effectiveness  in  detecting 
chemical  warfare  agents  in  a  threat  reahstic  environment. 
Model  results  can  be  found  in  Holman  et  al.  (2007). 

The  JBSDS  modeling  and  simulation  was  conducted 
by  a  team  that  included  JHU  APL,  the  Joint  Project 
Manager  for  Biological  Defense,  Air  Eorce  Operational 
Test  and  Evaluation  Center,  and  the  Army  Test  and 
Evaluation  Command.  The  model  was  based  on  the 
capabihty  of  the  system  to  re-analyze  “playback”  data 
from  previous  trials.  Some  of  the  parameters  in  the 
system  “playback”  are  then  changed  from  parameters  that 
are  indicative  of  simulant  to  parameters  that  are  indicative 
of  biological  warfare  agent.  The  basic  playback  uses  field 
DT  or  OT  data.  The  agent  unique  parameters  are 
measured  in  a  laboratory  during  a  DT  event. 

These  modeled  results  are  the  best  source  of 
performance  data  to  evaluate  the  detection  perfor¬ 
mance  of  JBSDS.  The  modeling  and  simulation 
provided  a  means  to  integrate  DT  and  OT  in  an 
evaluation  that  provided  information  on  the  effective¬ 
ness  in  detecting  and  discriminating  biological  warfare 
agents  in  a  threat  realistic  environment.  This  process  is 
described  in  Shirakawa,  Russell,  and  Holman  (2008) 
and  Przybylowicz  et  al.  (2003). 

Paradigm  3  modeling  and  simulation  provides  an 
opportunity  and  means  to  integrate  DT  and  OT  results 
to  support  an  integrated  evaluation  of  how  a  system 
performs  when  challenged  with  agent. 

Conclusion 

Biological  and  chemical  warfare  agents  cannot  be 
released  into  the  environment  during  OT,  and  they  can 
only  be  tested  in  DT  laboratory  containment  cham¬ 
bers.  Hence,  the  only  way  to  obtain  chemical  and 
biological  defense  system  operational  relevant  evalua¬ 
tions  is  to  integrate  DT  and  OT.  □ 
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